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Description 

Video is being generated at an ever-increasing rate everyday. In order to use information from such video reposi- 
tories efficiently and effectively, they must be properly indexed into a database. The most fundamental and important 
5 task in this process is to parse the video into an appropriate set of units, which are known as shots. Most existing 
approaches are based on preset thresholds or improper assumptions which reduce their applicability to a limited range 
of video types. 

The following description of the present invention also includes a mathematical formulation of the problem, a fast 
and robust implementation of the mathematical formulation, and a cut browsing apparatus. Performance has been dem- 
10 onstrated on over 149,000 video frames that include various video types such as sports and movies. 

Sources of videos include defense/civilian satellites, scientific experiments, biomedical imaging, fingerprinting 
devices, and home entertainment systems. In order to use information from such video repositories efficiently and 
effectively, videos must be properly indexed into a database. Video indexing provides a fast and accurate way to access 
desired video data based on its contents. The most fundamental and important task in this process is to parse the video 
75 into an appropriate set of units, which are known as shots. 

A shot in video refers to a contiguous recording of one or more video frames depicting a continuous action in time 
and space. In a shot, camera could remain fixed, or it may exhibit one of the characteristic motion such as panning, tilt- 
ing, or tracking. For most videos, shot changes or cuts are created intentionally by video/film directors. In the early 
years, they were done on a splicer and an optical printer, while shot lists were kept on paper records, the count sheets. 
20 Since 1985, most shot changes have been generated using modern editing machines. Information on each individ- 
ual shot is retained electronically on an Editing Decision List (EDL), which can be indexed directly into a database. How- 
ever, for most videos/films that were produced before the invention of such machines, this information, which was 
recorded on paper, may no longer be accessible. This is certainly true for home videos and raw sports films, since cuts 
are generated by turning the camcorders/film cameras on and off. In either case, cuts will have to be detected from the 
25 video through manual or automatic means. 

Unlike what is claimed in the literature, the isolation of shots in a video is not a trivial task considering the complex- 
ity of a scene and the efficacy of modern editing technologies. 

The transition from one shot to another may include visually abrupt straight cuts or camera breaks. It may also 
include such effects as fades, dissolves, wipes, flips, superimposures, freeze or hold frames, flop-overs, tail-to-head 
30 reverses, blow-ups, move-ins, repositions, and skip frames. See, for example, B. Balmuth, Introduction to Film Editing, 
Focal Press, 1989. 

Since the purpose of video indexing is to aid the retrieval of desired video clips from a database, it is important that 
a 100% recall rate and maximum precision rate be maintained. Here, a recall rate is defined as the percentage of the 
actual cuts that are detected and a precision rate, the percentage of the detected cuts that are actual cuts. 

35 ■ * Most existing approaches deal with very specific cases of few of the above transition types and even then a 100% 
recall rate is never achieved. In addition, they are often tested on few types of videos and few thousand video frames, 
insufficient to make any realistic judgment on the performance of individual algorithms. Most existing algorithms can not 
be implemented near video rate, imposing a serious constraint on the range of applications they can be used for. More 
importantly, they are often based on preset thresholds or improper assumptions which reduce their applicability to a lim- 

40 ited range of video types. 

Reference is also made to the following US. patent applications as containing material of close interest to the 
present application: U.S. Patent Application Serial No. 08/221,227 filed March 31, 1994 in the name of Arman et al.; 
U.S. Patent Application Serial No. 08/221 ,225 filed March 31 , 1994 in the name of Arman et al.; U.S. Patent Application 
Serial No. 08/221,221 .filed March 31, 1994 in the name of Arman etal.; U.S. Patent Application Serial No. 08/343,793 

45 filed November 22, 1994 in the name of Arman et al.; U.S. Patent Application Serial No. 08/346,453, filed November 29, 
1994 in the name of Benson et al. 

It is herein recognized that a good cut detection method is one that can 

— provide the maximum recall and precision rate for visually abrupt cuts and camera breaks, 

so — detect cuts, from reading the video to the output of shot change frames, close or near the video rate, 

— considers the nonstationary nature of the cut detection problem, 

— have a feedback mechanism to achieve a 100\% recall rate, work on a variety of videos and big number of video 
frames, and 

— is independent of the encoders and different encoding algorithms, when applied to compressed video. 

55 

In accordance with an aspect of the invention, cut browsing and editing apparatus for video images comprises a 
video player, a a video browser, coupled to the video player, and a cut browser coupled to the video browser. 

In accordance with another aspect of the invention, the video player and the cut browser present only raw video 
material. 
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In accordance with another aspect of the invention, the cut browser presents material at a different level of abstrac- 
tion to that presented by the video player and the video browser presents interpreted information about shots in the 
video images. 

In accordance with yet another aspect of the invention, cut browsing apparatus for video images, comprises a video 
s player, a video browser, coupled to the video player, a cut browser coupled to the video browser, and a display, coupled 
to the foregoing, the display providing a three-window display on a screen. 

In accordance with yet another aspect of the invention, each window of the three-window display is respectively 
allocated to one of the ideo player, video browser, and the cut browser. 

In accordance with still another aspect of the invention, the windows respond synchronously activities including 
w computer mouse pointer control, scrolling, and high-lighting. 

In accordance with still another aspect of the invention, the video browser displays detected shots using represent- 
ative frames (Rframes), and the cut browser displays a cross sectional image made from an entire video sequence. 

In accordance with still yet another aspect of the invention, a user can select, as by mouse clicking, a point on the 
cross sectional image on the cut browser so as to view a corresponding image in raw video playback on the video 
is player. 

In accordance with still yet another aspect of the invention, the cut browser permits editing by a user so as to break 
a shot into two shots and so as to merge any two shots into one, as by clicking a mouse. 

In accordance with a further aspect of the invention, the editing causes resegmenting of the cross sectional image. 

In accordance with still a further aspect of the invention, apparatus is included for presenting two cross-section 
20 images of the video images, a horizontal cross section image one is in a horizontal direction and a verical cross section 
image in a vertical direction of the video volume. 

In accordance with still a further aspect of the invention, cut browsing apparatus for video images, comprises a 
video player, a video browser, coupled to the video player, a cut browser coupled to the video browser, and a display, 
coupled to the foregoing, the display providing a three-window display on a screen, wherein each window of the three- 
25 window display is respectively allocated to one of the ideo player, video browser, and the cut browser. The windows 
respond synchronously activities including computer mouse pointer control, scrolling, and high-lighting. The video— 
browser displays detected shots using representative frames (Rframes), and the cut browser displays two cross-section 
images of the video images, a horizontal cross section image one in a horizontal direction and a verical cross section 
image in a vertical direction of the video volume. 
30 In accordance with yet a further aspect of the invention, each cross-section image is constructed by sampling one 
row (or column) from every image, and reducing the amount of information from a two-dimensional image to two one- 
dimensional image strips. 

In accordance with yet a further aspect of the invention, the horizontal and vertical cross-section images are com- 
bined into one image segmented into two bands according to a list of detected shots, whereby a level of abstraction is 
35 presented that is just enough to reveal whether there is a missed or misdetected shot. 

In accordance with still yet a further another aspect of the invention, cut browsing apparatus for video images com- 
prises a video player, a video browser, coupled to the video player, a cut browser coupled to the video browser, and a 
display, coupled to the foregoing, the display providing on a screen respective window displays for each of the video 
player, video browser, and cut browser, such that activities in all three are completely synchronized. 
40 In accordance with yet a further aspect of the invention, the windows respond synchronously activities including 
computer mouse pointer control, scrolling, and high-lighting. 

In accordance with still yet a further aspect of the invention, a user can select an image, as by mouse clicking, to 
view on the cut browser an image corresponding to a selected raw video image on the video player. 

The invention will be more clearly understood from the following detailed description of preferred embodiments, in 
45 conjunction with the Drawing, in which 

Figure 1 shows that a traditionally used, prior art, cut detection algorithm can be categorized as a process compris- 
ing four steps, data acquisition, difference metric collection, detection, and decision, wherein the dashed boxes 
indicate optional steps which can be found in some algorithms and wherein the delay circuit is a mechanism to uti- 
50 lize information from both the past and the future frames; 

Figure 2 shows four types of 3:2 pulldown; 

Figure 3 show a histogram of a typical inter-frame difference image that does not correspond to a shot change; 

55 

Figure 4 shows browsing apparatus in accordance with the invention; 

Figure 5(a) shows two cross sections in video volumes and Figure 5(b) shows a space-time image made up of two 
cross sections, useful in understanding the invention; 
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Figure 6 shows examples of cross section patterns in accordance with the invention, wherein Figure 6(a) shows 
visually abrupt cuts and Figure 6(b) shows gradual shot transitions, in black and white, and Figure 6(c) and 6(d) 
being the same figures in color; 

5 Figure 7 shows a flow diagram of a cut detection algorithm in accordance with the invention; 

Figure 8 shows a sub flow diagram of the difference metric collection step in accordance with the invention; 
Figure 9 shows a sub flow diagram of the adaptation step in accordance with the invention; 

10 

Figure 10 shows the membership function of contrast estimate in accordance with the invention; 

Figure 1 1 shows the membership function of motion estimate in accordance with the invention; 

75 Figure 12 shows the membership function of K a which is proportional to significance level in accordance with the 
invention; 

Figure 13 shows a sub flow diagram of the detection step in accordance with the invention; 

20 Figure 14 shows an example of trend estimation, wherein Figure 14(a) shows that a type I observation outlier is 
detected and Figure 14(b) shows that a type II observation outlier is detected; and 

Figure 15 shows cut browser apparatus incorporating the invention, 15(a) being in black and white and 15(b) being 
the same figure in color. 

25 

This present invention relates to an automatic cut detection method and a cut browsing apparatus that satisfy all 
criteria listed above. The cut detection is formulated as a non-stationary time series outlier detection problem. 

The efforts on cut detection can be traced back to D. Coll and G. Choma, "Image Activity Characteristics in Broad- 
cast Television," IEEE Trans, on Communications, Vol. 26, pp.1201-1206, 1976, where the authors perform an exten- 

30 sive experimental study of frame difference signals using four different types of videos, a football game, a drama show, 
a talk show, and a cartoon. The authors divide each uncompressed video frame into 8x8 blocks, each of which is rep- 
resented by its average grayscale value. In their study, the average of the magnitudes of the differences between cor- 
responding blocks of consecutive frames is used as the difference metric. Coll and Choma show that a threshold 
determined experimentally can detect shot changes that match the accuracy of a human observer. More scene change 
"35 detection algorithms based on uncompressed video can also be found in the computer vision literature. See for exam- 
ple, I. Sethi, V. Salari, and S. Vemuri, "Image Sequence Segmentation using Motion Coherence. "Proc. First Interna- 
tional Conference on Computer Vision}, pp.667-671 , 1987; Y. 2. Hsu, H.-H. Nagel, and G. Pekers, "New Likelihood Test 
Methods for Change Detection in Image Sequences," CVGIP. 26, pp. 73-106. 1984. I. Sethi, V. Salari, and S. Vemuri, 
"Image Sequence Segmentation using Motion Coherence," Proc. First International Conference on Computer Vision, 

40 pp.667-671, 1987. 

Digital video can be stored and transmitted in compressed form. See Joint Photographic Experts Group (JPEG); 
ISO/IEC JTC1 SC29 WG1. JPEG}, ISO/IEC 10 918; Moving Picture Experts Group (MPEG) ISO/IEC JTC1 SC29 
WG11, MPEG-1 , ISO/IEC 11 172, and MPEG-2, ISO/IEC 13 818, and International Telegraph and Telephone Consult- 
ative Committee (CCITT) CCITT, Recommendation H.261 , Video Codec for Audiovisual Services at px64 kbits/s, Dec. 
45 1990. 

Performing cut detection as well as image processing on compressed video saves unnecessary decompression- 
compression effort. This idea led to many efforts in pursuing solutions that can process compressed video directly, as 
disclosed by F. Arman et al. in U.S. patent applications serial No. 08/221,227 and 08/221,227, presently pending. 
Arman et al. have developed a scene change detection algorithm for JPEG and movie-JPEG video, where a subset of 
so Discrete Cosine Transform (DCT) coefficients is used to characterize the underlying frame. Unfortunately, full DCT coef- 
ficients are difficult or impractical to obtain in MPEG or H.261 video without full scale decoding. This is because motion 
vectors are quantities in the spatial domain, while DCT coefficients are quantities in the frequency domain. 

Other researchers have proposed to use motion vectors directly either to filter out scene change frames or to detect 
scene cuts See. for example, K-C H. Liu and G. L. Zick, "Scene Decomposition of MPEG Compressed Video," SPIE 
55 Vol. 2419, Digital Video Compression Algorithms and Technologies, pp.26-37, 1995; J. Meng, Y. Juan, and S.-F. Chang, 
"Scene Change Detection in a MPEG Compressed Video Sequence," SPIE Vol. 2419, Digital Video Compression 
Algorithms and Technologies, pp.1 4-25, 1995; and H. Zhang, C. Y. Low, and S. W. Smoliar, "Video Parsing and Brows- 
ing UsingC mpressed Data," Multimedia Tools and Applications, 1,pp.89-111. 1995. 

They are often based on the ratio of number of forward-predicted macro blocks and the total number of macro 



4 



EP 0 780 844 A2 



blocks. Since there is no standard criterion in determining whether a certain macro block should be inter-coded (tem- 
porarily predicted from a pre/ious reconstructed picture) or intra -coded (like a baseline JPEG picture) during the encod- 
ing process, this approach is very sensitive to different encoders and types of encoding algorithms. 

Most cut detection algorithms in the literature, whether or not they take uncompressed or compressed video data. 

s can be categorized, as a process comprising four steps: data acquisition, difference metric collection, detection, and 
decision, as illustrated in Figure 1. The dashed boxes indicate optional steps, which are found in some algorithms. The 
delay circuit is a mechanism to use information from both past and future frames. 

During the data acquisition stage, compressed video based approaches often extract DCT coefficients or motion 
vectors. See, for example, the above-cited application of Arman et al.; H. Ishikawa and H. Matsumoto, "Method for 

10 Detecting a Scene Change and Image Editing Apparatus," European Patent 0-61 5-245-A2, filed July 3, 1994; H.-C. H. 
Liu and G. L. Zick, "Scene Decomposition of MPEG Compressed Video," SPIE Vol. 2419, Digital Video Compression 
Algorithms and Technologies, pp.26-37, 1995. There are a total of 64 DCT coefficients, including one DC (the base fre- 
quency in the DCT coefficients) term and 63 AC (the higher frequencies in the DCT coefficients) terms. The amount of 
decoding varies among different algorithms. Some only decode DC terms, see, for example, J. Meng, Y Juan, and S - 

15 R Chang, "Scene Change Detection in a MPEG Compressed Video Sequence," SPIE Vol. 2419, Digital Video Com- 
pression Algorithms and Technologies, pp.14-25, 1995, while others only extract the number of forward-coded and 
backward-coded motion vectors, see, for example, H.-C. H. Liu and G. L. Zick, "Scene Decomposition of MPEG Com- 
pressed Video," SPIE Vol. 2419, Digital Video Compression Algorithms and Technologies, pp.26-37, 1995. 

Most such algorithms try to avoid full scale decoding in order to be computationally efficient. For approaches that 

20 use uncompressed video data, each digitised image is often smoothed or sub-sampled; see, for example, P. Aigrain and 
P. Joly, "Trie Automatic Real-Time Analysis of Film Editing and Transition Effects and its Applications," Computer and 
Graphics. Vol. 18, No. 1. pp.93-103. 1994. 

Once video data is acquired, except for those algorithms that use motion vector information explicitly, most in the 
literature collect one difference metric per frame. Such difference metric, in general, can be categorized into pixel-based 

25 and distribution-based types. 

In the pixel-based category, the most popular difference metric is the sum of squared difference, namely _ 

drYV'.-f'-y 

so r " /y 



where f*,y represents the intensity value at pixel location (i, j) in frame t. An alternative choice for less computation is the 
sum of absolute difference 

35 

f ^ ij a 

40 

Both difference metrics are sensitive to large camera moves, leading to numerous false positives in cut detection. 
To overcome this problem, many researchers either smooth video frames, or down sample them, before taking the pixel 
based differences. This additional step often improves the precision rate of cut detection algorithms significantly. 

Distribution-based difference metrics, on the other hand, are less influenced by camera moves. A likelihood meas- 
45 ure was first proposed in the late 70's based on the intensity mean and variance of the current and the previous frame. 
See Y. Z. Hsu, H.-H. Nagel, and G. Pekers. "New Likelihood Test Methods for Change Detection in Image Sequences," 
CVGIP, 26, pp. 73-106, 1984. If \x t and a, are used to represent the intensity mean and variance of the frame t respec- 
tively, the likelihood based difference metric can be defined as 

d t = i ^ (1) 

55 Other existing distribution-based difference measure are based on histograms. The image histograms of the f h frame 
is denoted by H t . Let H t (j) be the pixel counts in bin j of the histogram H t . If the total number of bins is N, one can 
compute either: 
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w 



20 



25 



50 



55 



7=1 

or 



d m " '>H,(j)-H,.M> Z 



dt ^m±^ml (3) 



Either one of them can be used as a distribution-based difference metric. The bin size N in either measure affects 
the sensitivity of the cut detection algorithm. The smaller the bin size is, the more sensitive the histogram measure will 
be. In most existing methods, a fixed bin size is used throughout all intensity ranges. 
15 Recently, a third type of distribution-based difference metric has been proposed. See I. K. Sethi and N. Patel t "A 
Statistical Approach to Scene Change Detection," SPIE Vol. 2420, Storage and Retrieval for Image and Video Data- 
bases III, pp. 329-338, 1995. This metric is based on the empirical distribution function (EDF) of the previous and cur- 
rent frames. It is called the Kolmogorov-Smirnov test metric. 



d t =max}EDF t U)-EDF t .,{j)\ (4) 

In order to compute the empirical distribution function, one first constructs the histogram of each individual video 
frame. Assume that the histogram for frame t is represented by { H t {j)\j^,N }. EDF t {j) , which represents the cumu- 
lative distribution probability of the jth intensity value in frame t, can then be defined by 

30 EDF t U)=EDF t (H) + ^ , j = 2,N 

where M is the total number of pixels in the image. 

After the difference metric is collected for each frame, most algorithms simply apply a preset threshold to this time 
-35 - series of difference metrics. The algorithm compares the value of difference metric with a global threshold. If it is above 
this preset number, a shot change is signaled. Since it is difficult to find such a threshold for an entire video, some 
researchers have also proposed the use of two thresholds. See, for example, H. Zhang, A. Kankanhalli, and S. W. Smo- 
liar, "Automatic Parsing of Full-Motion Video," ACM Multimedia Systems, 1, pp.10-28, 1993. 

Recently, rank-based detection schemes have become very popular. See, for example, P. Aigrain and P. Joly. "The 
40 Automatic Real-Time Analysis of Film Editing and Transition Effects and its Applications," Computer and Graphics, Vol. 
18, No, "l.pp.93-103, 1994. 

Since global thresholds are hard to find, the idea is that thresholds should be applied only to difference metrics in 
a local temporal neighborhood. This local temporal neighborhood approach is indicated as the dashed box attached to 
the detection step in Figure 1 . In order for this to work, a new difference metric is computed in the detection step from 
45 every local temporal window centered around the current frame. Denote d t as the difference metric for frame t in the 
form of sum of squared differences (or sum of absolute differences) and assume that the size of the temporal window 
is2N+1. !f,the2N+1 observations dj , i=t-N t t+N are ordered d^<d z <* • *<d ZN ^ , the new metric d t is often com- 
puted in one of the following three ways. 



or 



3 t =0 if C? t *<3 2Jm ; 

a t =J^l otherwise Eq ' 5 

V2N 



d f =0/7d^d 2W+1 ; Eq.6 
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rf f =cf 2 A/+i '^2N otherwise 



or 



10 



a t =0 if de*^; 
3 e - otherwise E <I- 7 



The preset threshold wili then be applied to this sequence of new metrics d t . Most algorithms that use Equation 5 or 
Equation 7 favor N = 2 and a number between 3 and 4 as the best threshold choice. 

All detection schemes discussed so far are based on preset thresholds and they do not treat cut detection as a 

15 hypothesis testing problem. On the other hand, there are a few cut detection algorithms, that are based on hypothesis 
testing paradigms. All such algorithms can be categorized into two different types. The first formulation views a 
sequence of difference metrics as a set of samples drawn from a single known distribution, if they are derived from 
images within the same shot. The second formulation views them as a set of samples drawn from a single but unknown 
distribution. Note here that the difference metric could be computed from each individual frame or from any two neigh- 

20 boring frames. These formulations transform the cut detection problem into the following two problems respectively: 

Case 1 : Can one disprove, to a certain required level of significance, the null hypothesis that difference metric is a 
sample drawn from a known distribution? 

25 Case 2: Can one disprove, to a certain required level of significance, the null hypothesis that the previous and the 
current difference metric are samples drawn from the same distribution? 

In both cases, either pixel -based or distribution-based difference metrics can be used. However, the underlying dis- 
tribution in the second case does not have to be known in advance. Each formulation will next be explained in more 
30 detail. 

Regarding Case 1 , work on modeling frame difference signals can be traced back to the 1960's. Seyler first studied 
the nature of frame difference signals and showed that the gamma distribution provides a good fit to the probability den- 
sity function of the fraction of pixels whose change is above a threshold. See A. J. Seyler, "Probability Distributions of 
Television Frame Differences," Proceedings of I.R.E.E. Australia, pp.355-366, November 1965. An analytic expression 
35 - is derived for the probability density functions approximating the experimentally recorded frame difference distributions 
and is simply the following: 



40 



h(D)-(T(a)^)- 1 If t " x exp ( - -5 ) 

where D is the fraction of pixels whose change is above a threshold and the parameters a and p are given by the com- 
puted mean ^ and standard deviation of the experimental distribution, as 



45 M- 2 0 _2 



50 

It is however not clear how sensitive this conclusion is to the choice of thresholds. 

An alternative mode! for frame difference signals is that the corresponding difference d fj {=f n -f n ^) at pixel location 
follows a zero-mean Gaussian distribution with variance ay . The unknown parameter c^'carl be estimated from 
the image sequence directly To simplify the model, it is often assumed that the random variables are i.i.d. (independ- 
55 ently identically distributed), therefore 

<jjj=a for all (/,/) 

Following this, it is shown in T. Aach, A. Kaup, and R. Mester." Statistical Model-Based Change Detection in Moving 
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10 



20 



Video." Signal Processing, 31, pp 165-180, 1993 that 



2 



obeys a x 2 distribution with as many degrees of freedom as the number of pixels in the image. 
djj can also be modeled as obeying a Laplace distribution, i.e., 

P(<V=|exp{-y!(/ir!} 



where y=— and a 2 is the variance of this probability density function. It is then shown in Aach et al. op. cit. that 
15 p(2?.^djf) follows a x 2 distribution with twice as many degrees of freedom as the number of pixels in the image. 

Among distribution -based difference metrics, the difference metric constructed using equation (1) is based on the 
assumption that images are samples drawn from N([L t ,o t ) . This metric is essentially the maximum likelihood ratio of 



L{H X ) 
L(H Q ) 



where L(x) is the likelihood of event x and 
25 tf 0 :frames M and t come from the same distribution A/(n M ,a M ) :frames M and t come from different distributions 
/V(u Ml a M )and/V(^,a f ). 

For the difference metric constructed using equation (2) and (3), it can also be shown that both satisfy the x 2 dis- 
tribution. See J. D. Gibbons and S. Chakraborti, Nonparametric Statistical Inference, Marcel Dekker, Inc., 1992 The % 2 
test detects whether an observed frequency distribution in a sample arise from the assumed theoretical distribution. 
30 Such distributions could be a true binomial, Poisson, normal, or some known type of distribution in the population. Usu- 
ally, the parameters of this distribution is not known. It may be shown that, if s parameters are estimated by the method 
of maximum likelihood, the limiting distribution of 



is that of x 2 with /V-s-1 degrees of freedom. 
40 For Case 2, the most commonly used statistical approach of this type is the Kolmogorov-Smirnov test. The Kol- 
mogorov-Smirnov test is concerned with the agreement of two sets of observed values and the null hypothesis is that 
two samples come from populations with the same distribution function F(x) . It can be shown that the test statistic d t 
as in Equation (4) satisfies 

Prob(d t >K a ^a 

where K a represents a constant depending on the level of significance a and is available in tabular form from most sta- 
50 tistics books. N is the size of the image the statistic is collected upon. 

Unlike the test discussed just above, the Kolmogorov-Smirnov test does not assume any it a priori information 
about the type of distribution functions. 

The decision stage is often set up to eliminate detected cuts that are false positives due to flash light or slow/fast 
motion in the scene. The list of detected cut frames is stored in a delay memory as shown by the dashed box attached 
55 to the decision step in Figure 1. The criterion usually states that the minimum distance between two cuts has to be 
greater than a preset number. 

Most existing approaches are based on preset thresholds or improper assumptions which reduce their applicability 
to a limited range of video types. For example, incorrect assumptions have frequently been made about how shots are 
connected in videos, ignoring the realities of how films/videos are produced and edited. The nonstationary nature of the 
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cut detection problem has often been ignored, as well as the fact that many videos are converted from films in the 
design of their cut detection algorithm. 

In the following section, the cut detection problem will be examined from several different viewpoints. Since most 
shots and shot changes are created intentionally by film/video directors, it is very important to understand techniques 
s commonly used by fSm/video production and editing professionals. This will provide an idea what types of shots and 
cuts could be present in films/videos. The commonly used techniques to convert films to videos will also be reviewed. 
This understanding is very important for developing a robust cut detection algorithm. Finally, different assumptions 
made by existing methods will be explained and a solution in accordance with the invention will be described. 

In the field of film/video production, a few shot classifications are established for directors, camera operators, and 
w editors so that they can use the same language when talking about particular shots. There are three typical shots 
present in most f ilms/videos. 

— static shots, which are taken without moving the camera 

15 The various types of static shots fall into five categories: 

close-up, close, medium, full, and long. Different static shots induce different amounts of change from frame to 
frame. For example, an object's movement in a close shot will produce more significant changes in comparison with 
a medium or long shot. 

20 

— camera moves 

The various types of camera moves include zooms, tilts, and pans, or a mixture of the above. The change 
from frame to frame is a function of the speed at which camera moves. For example, a fast camera pan will 
induce more significant changes in comparison with a slow camera pan. 

25 

— tracking shots, where the camera is moved during the shot 

The most popular tracking shots include walking shots, dolly shots, and crane shots. All tracking shots 
have the tendency to induce unsteadiness, in particular when the object being tracked moves too fast. It is obvi- 
ous that the amount of change from frame to frame is also a function of how steadily the camera tracks the 
30 object. 

Since camera/object motion in different shot types result in different amounts of intensity change, the criteria to 
detect cuts should be different when different types of shots are processed. Otherwise, many false positives and false 
negatives could occur and thus reduce the recall and the precision rate. Unfortunately, in the type of problem this inven- 
ts ■ tion disclosure addresses, shot types are not known in advance. How to adjust the detection criteria to different shot 
types becomes one of the most important tasks in cut detection. 

Another important issue to be looked at is the fact that many videos are converted from films, because the two are 
played at different frame rates. A common conversion process, called a 3-2 pull down, is to make every other film frame 
a little bit longer. This process is often performed by a telecine machine and may affect many cutdetection algorithms. 
40 As shown in Figure 2, there are actually four different ways that the 3:2 transfer can take place, as follows. In Figure 2, 
W, X, Y, Z are four film frames, each of which contains of two fields. By manipulating these field images, one can con- 
struct five video frames, (upper left) starting on field 1 , alternating 3 fields (upper right) starting on field 1 , alternating 2 
fields (lower left) starting on -field 2, alternating 3 fields (lower right) starting on field 2, alternating 2 fields. 

Thus: 

45 

1. starting on field 1, alternating 3 fields, then 2..3..2..3..2. 

2. starting on field 1, alternating 2 fields, then 3..2..3..2..3. 

3. starting on field 2, alternating 3 fields, then 2..3..2..3..2. 

4. starting on field 2, alternating 2 fields, then 3..2..3..2..3. 

50 

In any case, certain video frames are made up of two fields with totally different (although consecutive) pictures in 
them. As a result, the digitizer can only take one field from each video frame in order to maintain good picture quality. 
This will result in duplicated frames and almost zero inter-frame differences at five frame intervals in all four cases of 
3:2 pulldown. For cutdetection methods that are based on pixel-based difference metrics, this may lead to numerous 
55 false positives. That is the reason why many existing algorithms based their detection on difference metrics collected in 
a five or seven frame interval centered around the current frame. See, for example, P. Aigrain and P. Joly, "The Auto- 
matic Real-Time Analysis of Film Editing and Transition Effects and its Applications," Computer and Graphics, Vol. 18, 
No. 1, pp. 93-1 03, 1994; H. Dubner, "Video Logging System and Method Thereof," International Patent Classification 
H04N9/79,5f76. Application Filed November 1993; T. Koga, "Accurate Detection of a Drastic Change Between Succes- 
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sive Pictures," US Patent 5,032,905, July 16, 1991. A similar problem occurs in animated videos such as cartoons 
except it produces almost zero inter-frame differences in as often as every other frame. 

Most shot transitions are created by film/video directors. Before the invention of modem editing machines, it is often 
done on a splicer and an optical printer. 
5 The following are the most common shot transition and special effect types that can be created by such devices: 

— fade in, the incoming scene gradually appears, starting from dark 

— fade out, the outgoing scene gradually disappears, ending in black 

— dissolve, comprises an outgoing scene, which fades out, and an incoming scene, that fades in and overlaps the out- 
re going scene 

— the incoming scene is used to literally wipe off the outgoing scene by means of a hard-edged or soft-edged line, as 
the incoming scene becomes solely and completely visible 

— flip, a widening black stripe either on both sides or on both tops of the picture frame, that squeezes out the outgoing 
scene and simultaneously brings in the incoming scene, thus creating an extremely fast turnover, or revolving, 

15 effect 

— freeze frame, a particular frame is selected to repeat for any desired length for dramatic or comedic effect 

— flop-over, an effect often used to correct misdirection 

— tail-to-head reversal, the tail, or end, of the cut becomes the head and the head, the tail. 

— skip frames, the elimination of frames to speed up the action 
20 — double printing, the duplicate of frames to slow down the action 

— blow up, the blow up of frames to eliminate certain unwanted objects 

— move-in, ending in a dissolve to another moving shot 

— reposition, adjusting the original angle of a shot 

— superimposure, the overlapping of two or more scenes which, unlike those in a dissolve, are simultaneously of con- 
25 stant relative strength. 

— montage, any sequence of superimposures 

Modern computer technology allows even more ways to create shot transitions. The following are some examples. 

30 — band wipe, the incoming scene is revealed under the outgoing scene by horizontal or vertical bars 

— barn doors, the incoming scene is revealed under the outgoing scene from the center outwards 

— center merge, the outgoing scene splits into four parts and slides to the center to reveal the incoming scene 

— center split, the outgoing scene splits into four parts and slides to the corner to reveal the incoming scene 

— checkerboard, two sets of alternating boxes wipe to reveal the incoming scene 

35 — cross stretch, the incoming scene stretches from an edge as the outgoing scene shrinks 

For transitions made through visually abrupt straight cuts or camera breaks, the cut detection problem is a relatively 
well-defined task. However, detecting other types of transitions such as fades, dissolves, wipes, flips, superimposures, 
freeze or hold frames, flop-overs, tail -to-head reverses, blow-ups, move-ins, repositions, and skip frames, may not be 

40 as straightforward. As a matter of fact, most scene transition can be any length an editor wishes, as long as it is con- 
sistent with the style, mood, and pacing desired at that moment in the picture. Therefore, the collected difference met- 
rics associated with these shot transitions are often indistinguishable from those associated with gradual camera 
moves, unless semantic levelinformation such as the image motion pattern is also considered. See, for example, H. 
Zhang, A. Kankanhalli, and S. W. Smoliar, "Automatic Parsing of Full-Motion Video," ACM Multimedia Systems, 1, 

45 pp.10-28, 1993. 

Many existing methods try to model various varieties of shot transitions. See, for example, P. Aigrain and P Joly, 
"The Automatic Real-Time Analysis of Film Editing and Transition Effects and its Applications," Computer and Graph- 
ics, Vol. 18, No. 1 , pp.93-103, 1994; A. Hampapur, R. Jain, and I Weymouth, "Digital Video Segmentation," Proc. ACM 
Multimedia Conference, pp.357-363, 1994; J. Meng, Y. Juan, and S.-F. Chang, "Scene Change Detection in a MPEG 

so Compressed Video Sequence," SPIE Vol. 2419, Digital Video Compression Algorithms and Technologies, pp. 14-25, 
1995; JPEG ISO/IEC JTC1 SC29 WG1, JPEG, ISO/IEC 10 918; T. Koga, "Accurate Detection of a Drastic Change 
Between Successive Pictures," U.S. Patent 5,032,905, July 16, 1991 ; MPEG ISO/IEC JTC1 SC2. B.-L. Yeo and B. Uu, 
"Rapid Scene Analysis on Compressed Video," To appear in IEEE Trans, on Circuits and Systems for Video Technol- 
ogy, 1995. They often assume that both the incoming and outgoing shots are static scenes and the transition only lasts 

55 no longer than half a second. This type of model is too simplified to model gradual shot transitions that are often present 
in films/videos. 

Another assumption researchers often make is that the frame difference signal computed at each individual pixel 
can be modeled by a stationary independently identically distributed random variable which obeys a known probability 
distribution such as the Gaussian or Laplace. See, for example, H. Zhang, A. Kankanhalli, and S. W. Smoliar, "Auto- 
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matic Parsing of Full-Motion Video," ACM Multimedia Systems, 1, pp.10-28, 1993. This assumption is generally not 
true, as shown in Figure 3 which shows the histogram of a typical inter-frame difference image that does not correspond 
to a shot change. Note that the shape changes as camera moves (left) slowly versus (right) fast. Neither a Gaussian 
nor a Laplace fits both curves well. A Gamma function fits the left curve well, but not the right curve. 

s This fact invalidates cut detection approaches that are based on x 2 tests since these tests are derived statistically 
from the above assumption. See, for example, G. Casella and R. L Berger, Statistical Inference, Duxbury Press, 1990. 

In addition, existing methods assume that time-series difference metrics are stationary, completely ignoring the fact 
that such metrics are highly correlated time signals. 

It is herein recognized that pixel-based and distribution-based difference metrics respond differently to different 

io types of shots and shot transitions. For example, the former is very sensitive to camera moves, but is a very good indi- 
cator for shot changes. On the other hand, distribution-based metrics are relatively insensitive to camera and object 
motion, but can produce little response when two shots look quite different but have similar distributions. It is an object 
of the present invention to combine both measures in cut detection. 

Unlike existing methods which have no notion of time series nor non-stationarity, the present invention treats a 

15 sequence of difference metrics as nonstationary time series signals and models the time trend deterministically. The 
sequence of difference metrics, no matter how they are computed, are just like any economic or statistic data collected 
over time. In this view, shot changes as well as the 3:2 pull-down process will both create observation outliers in time 
series, while the gradual shot transition and gradual camera moves will produce innovation outliers. Fox defines the 
observation outlier to be the one that is caused by a gross error of observation or recording error and it only affects a 

20 single observation. See A J. Fox. "Outliers in Time Series," Journal of the Royal Statistical Society, Series B, 34, pp. 
350-363. 1972. 

Similarly, the innovation outlier is the one that corresponds to the situation in which a single "innovation" is extreme. 
This type of outlier affects not only the particular observation but also subsequent observations. A typical model that 
represents observation outliers (occurs at f=q ) is 

25 

d t =f(d t . r d t _ r ^ t * • -,d t )+u t if teq Eq. 8 

d r =f(d f _ r ,tf f , r+1 , • • • ,tf f )+u f + A otherwise 
30 where 1 1 represents the time index, A is the outlier, f(d t _ n d t . r ^, • • • ,d t ) models the trend in the series, and 



,..35 



r=1 



(f=p+1, • • *,n) 



Eq. 9 



40 



In Equation (9), a r are autoregressive parameters and the {z t } are independently A/(0,cr| ) (zero mean normal dis- 

ition with variance a I ). 

The model for innovation outliers is 

p 

d t =f(d t . r d t ^,- • • ,d t ) + £a f d,. f +A f +z, Eq. 10 

r=1 



45 

where a r and [z t ] are defined as for equation (9) and the outlier A, affects d t and, through it, subsequent observations 
d t+ iA+2> ' # m >d n - 

There are standard methods in the literature which detect both outliers. See, for example, B. Abraham and A. 
Chuang, "Outlier Detection and Time Series Modeling," Technometrics, Vol. 31, No. 2, pp.241 -248, May 1989; A. J. 
so Fox, "Outliers in Time Series," of the Royal Statistical Society, Series B, 34, pp. 350-363, 1972; L. K. Hotta and M. M. 
C. Neves, "A Brief Review of Tests for Detection of Time Series Outliers," ESTADISTICA, 44, 142, 143, pp. 103-148, 
1992. 

These standard methods, however, can not be applied to the cut detection probi em directly as yet for the following 
three reasons. First, most methods require intensive computation for example, least squares, to estimate time trend and 
55 autoregressive coeff icierrts. This amount of computation is generally not desired. Second, the observation outliers cre- 
ated by slow motion and the 3:2 pull-down process could occur as often as one in every other sample, making the time 
trend and autoregressive coefficient estimation an extremely difficult process. Finally, since gradual shot transitions and 
gradual camera moves are indistinguishable in most cases, location of gradual shot transitions requires not only detec- 
tion of innovation outliers but also an extra camera motion estimation step. 
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In the solution in accordance with the present invention, a zero-th order autoregressive model and a piecewise-Iin- 
ear function are used to model the time trend. With this simplification, samples from both the past and the future must 
be used in order to improve the robustness of time trend estimation. More than half of samples are discarded because 
the observation outliers created by slow motion and the 3:2 pull-down process could occur as often as one in every 
5 other sample. Fortunately, these types of observation outliers are least in value, and therefore could be identified easily. 
After the time trend is removed, the remaining value is tested against a normal distribution A/(0,a) , which a can be 
estimated recursively or in advance. 

To make the cut detection method more robust, a further application is made of the Kolmogorov-Smirnov test to 
eliminate false positives. This test is chosen because it does not assume a priori knowledge of the underlying distribu- 
10 tion function. Trie traditional Kolmogorov-Smirnov test procedure compares the computed test metric with a preset sig- 
nificance level (normally at 95%). It has been used by some researchers to detect cuts from videos. See, for example, 
I. K. Sethi and N. Patel, "A Statistical Approach to Scene Change Detection," SPIE Vol. 2420, Storage and Retrieval for 
Image and Video Databases III, pp. 329-338, 1 995. This use of single preselected significance level completely ignores 
the nonstationary nature of the cut detection problem. It is herein recognized that the Kolmogorov-Smirnov test is prop- 
15 erly used only if it takes into account the nonstationary nature of the problem. In other words, the significance level 
should be automatically adjusted to different types of video contents. 

One way to represent video contents is to use measurement in both the spatial and the temporal domain together. 
For example, image contrast is a good spatial domain measurement, because the amount of intensity changes across 
two neighboring frames measures video content in the temporal domain. The adjustment should be made such that, 

20 

- the higher the image contrast is, (that is. the less the significance level), the more sensitive the cut detection mech- 
anism should be, and 

- the more changes occur in two consecutive images, the less sensitive the detection mechanism (that is, the higher 
the significance level) should be. 

25 

The traditional Kolmogorov-Smirnov test also can not differentiate the long shot from the close up of the same 
scene. To guard against such transitions, the present invention utilizes a hierarchical Kolmogorov-Smirnov test. In this 
hierarchical Kolmogorov-Smirnov test, each frame is divided into four rectangular regions of equal size and the tradi- 
tional Kolmogorov-Smirnov test is applied to every pair of regions as well as to the entire image. This test therefore pro- 
30 duces five binary numbers which indicate whether there is a change in the entire image as well as in each of the four 
sub-images. 

Finally, instead of directly using these five binary numbers to eliminate false positives, the test result is only used in 
a qualitative sense. The significance in test result of the shot change frame is compared against that of its neighboring 
frames. These modifications will become better understood in a later portion of the description. 

35 ' Despite many claims and attempts in the literature regarding detecting gradual shot changes, it is not possible to 
differentiate gradual shot transitions from gradual camera moves, since difference metrics associated with both cases 
are often indistinguishable. Zhang et al. present a hybrid algorithm that tries to differentiate gradual shot transitions from 
gradual camera moves based on camera motion analysis. See H. Zhang, C. Y. Low, and S. W. Smoliar, "Video Parsing 
and Browsing Using Compressed Data," Multimedia Tools and Applications, 1 . pp.89-1 1 1 , 1995. However, it should be 

40 noted that camera motion estimation has been an active area of research in computer vision for more than a decade 
and a reliable motion estimation algorithm is yet to be seen. In the context of the present invention, it is recognized 
herein that, since no single cut detection algorithm can deliver 100\% recall rate, a cut browsing apparatus is required 
to provide access to misdetected shots. This browsing apparatus could also be used to identify gradual shot transitions. 
How will the browsing apparatus provide information to users without the need to play back the entire video is the most 

45 challenging task in designing such an apparatus. 

In order to provide information to a user who can then identify missed and misdetected shots, this browsing appa- 
ratus should have the following three characteristics: First, it must present information different from what automatic cut 
detection algorithm extracts from the video. In addition, it should not try to present interpreted information to the user, 
since any such information could be the source of an error. Finally, the browsing apparatus should be fast and concise, 

so providing another level of abstraction such that the need to play back the entire video can be avoided. 

In the context of the present invention, browsing apparatus comprises a video player, a video browser, and a cut 
browser. See Figure 4. The cut browser forms the subject matter of a copending U.S. Patent Application Serial No. 
08/576,272, entitled APPARATUS FOR DETECTING A CUT IN A VIDEO, filed on even date herewith in the name of 
the present inventors, and whereof the disclosure is herein incorporated by reference. The video browser forms the sub- 

55 ject matter of U.S. Patent Application Serial No. 08/221,225, entitled REPRESENTING CONTENTS OF A SINGLE 
VIDEO SHOT USING RFRAMES, in the name of Arman et al., included, without its claims, in the present application 
as Appendix 1 . 

These three components provide three different levels of abstraction. Further details will be presented further on, 
but briefly, the cut browser presents two cross-section images, one is in a horizontal direction and the other in a vertical 
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direction of the video volume. Each cross-section image is constructed by sampling one row (or column) from every 
image, and reducing the amount of information from a two-dimensional image to two one-dimensional image strips as 
shown in Figure 5, in which (a) shows two cross sections in video volumes and (b) shows a space-time image made up 
of two cross sections. 

5 The horizontal and vertical cross-section images are combined, in the cut browser, into one image segmented into 
two bands according to the list of detected shots. This representation provides a level of abstraction just enough to 
reveal whether there is a missed or misdetected shot. 

This representation allows the user to easily search the content of the video, to decide if the cut detection method 
may have missed or misdetected a shot, and to create or eliminate cuts in preparation for a multimedia content. 

w This particular set of three levels of abstraction is chosen for the following reasons. First, the original video must be 
included because it is the only raw information available. However, no one can afford to search information from the 
original video directly because video playback is a very time-consuming task. That is a reason to include the represent- 
ative frames (Rframes) of each detected shot Any automatic cut detection method can be used to provide this level of 
abstraction. Second, since no cut detection method offers 100\% accuracy rate, there will always be missed or misde- 

is tected shots. In order to provide the user a clue so as to avoid being affected by the imperfection of any cut detection 
method (or any automatic method in the matter) in the search of desired video clip, additional information should be pro- 
vided that has the characteristics referred to above: 

— it must be completely different and independent from what the automatic cut detection algorithm extracts from the 
20 video 

— it should be raw information, but should not be the original video 

— "rt should be concise 

It is believed that the best choice is the cross-section image. A cross-section image is an image generated directly 

25 from the original video. The present implementation selects one in the horizontal and the other in the vertical direction 
of the video volume, although, in general, one can select any number of directions. The process of constructing a cross- 
section image is illustrated in Figure 5 . The horizontal (vertical) cross-section image is constructed by sampling the 
middle row (or the middle column) from every image and by collecting all samples over time. To provide a complete view 
of both cross-section images, they are combined into one image and then the image is segmented into two bands 

so according to the list of detected shots. 

This representation provides a level of abstraction which reveals the continuity of video frames. For example, if 
there is a missed shot which is the result of a visually abrupt cut or camera break, a clear discontinuous pattern is dis- 
cernible as shown in Figure 6(a). For cuts that are associated with gradual shot transitions such as dissolves, a blurred 
discontinuous pattern is apparent as shown in the bottom two pictures of Figure 6(b). 

35 Five components are utilized herein: a video player, a video browser, a cut browser, a Rframe generator, and a 
cross-section generator. Their inter-relationship is shown in Figure 4. In this figure, the video source can be either an 
analog or a compressed digital video and the automatic cut detection module can be any automatic cut detection 
method as long as it outputs a list of shots in frame numbers. The video player, the video browser, and the cut browser 
are three major components, which are supported by two other components, the Rframe generator and the cross-sec- 

40 tion generator. The activities in all components are completely synchronized. 

The Rframe generator takes the shot list as well as the original video and produces thumbnail pictures that repre- 
sent each shot. Whenever the shot list is updated, it updates the list of thumbnail pictures. The video player plays back 
the original video and accepts requests coming from either the video browser or the cut browser. The playback can start 
from any video frame. In order to locate the desired video frame, either the time code or the byte offset is used, depend- 

45 ing on whether the video source is an analog video or a digital compressed video respectively. 

The video player also has a VCR-like user interface which implements functions such as pause, play, slow motion, 
fast forward, and rewind. It also provide a shot jump capability allowing the user to skip shots which are detected by the 
automatic cut detection method. 

The video browser displays a list of thumbnail pictures representing each detected shot. It allows a user to quickly 

so glance through a video to find the clip of interest: The desired one will be found if it is among the list of detected shots. 
The video browser will properly position itself by taking information from either the video player or the cut browser. 
When a video is being played or the cross-section image is clicked, the representative frame of the associated shot will 
be highlighted. When the cross-section image scrolls, the list of thumbnail pictures will scroll accordingly. 

To provide information for missed or misdetected shots, the cut browser presents the cross-section image gener- 

55 ated from the raw video. In the foregoing description, the types of patterns that could appear in the cross-section image 
when there is a shot change where described and explained. These patterns of cross-section images provide useful 
information for manual identification of missed or misdetected shots. When a user clicks on the cross-section image, 
the cut browser maps the mouse location to a frame number and sends the request to both the video player and the 
video browser. The video will start playing from that frame and the associated representative frame in the video browser 
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will be highlighted. 

The foregoing exemplary embodiment has been described in terms of two cross-section images; however, it is 
within the contemplation of the invention that a plurality of such images, which may be greater than two, can be utilized. 
The user interface of the current invention is shown in Figure 15. In operation, initially, three windows display on the 
s screen. To search the content of the video, the user would examine each representative frame in the video browser to 
see whether it is the desired shot When the list of thumbnail pictures is being scrolled, the content of the cut browser 
will be updated to match the representative frame that is being highlighted. The user would clearly see if there is any 
missed shot between the current and the next shot. When the user is in doubt, they can simply click on the point which 
might start a missed shot in the cut browser to look at the content in the raw video. To decide if the cut detection method 
10 may have missed or misdetected a shot, the user could simply examine the cross-section image in the cut browser. If 
the user sees any pattern that might represent a missed or a misdetected shot, the user would then click on the point 
in the cut browser to look at the content of the raw video. 

The cut browser also provides the editing capability. This editing function is extremely useful far authoring content- 
indexed multimedia materials. It allows a user to break any shot into two, or merge any two shots into one, through a 
15 mouse or a button click. When this event happens, the cross-section image will be reorganized into two bands and the 
shot list is updated. The Rframe generator will then update the list of representative frames. 

For example, when ail video frames are in the same shot and all shots are correctly detected, a clear continuous 
pattern in this cross-section image is seen, as explained above in reference to the example shown in Figure 6. 

In addition, since these are also raw image data, no interpretation is presented and therefore no further error could 
20 be introduced. 

The problem analysis described in the previous sections is incorporated in the present inventive solution to the cut 
detection problem which allows a partitioning of a video into a set of shots comprising visually abrupt cuts or camera 
breaks. Figure 7 shows a flow diagram of a cut detection algorithm in accordance with the invention. 

Unlike prior art methods which use either pixel-based or distribution-based difference metric, the present invention 
25 integrates both types of information in a new detection scheme. The invention includes an adaptive engine to provide 
the detection scheme with the ability to automatically adjust to different types of video contents. The result from the 
automatic cut detection is presented to the user through a cut browsing apparatus, where three tasks can be performed: 
to search the content of the video, to decide if the cut detection method may have missed or misdetected a shot, and 
to create or eliminate cuts. 

30 The cut detection is herein formulated as a time series outlier detection problem. As shown in Figure 7, the system 
can take either uncompressed or compressed continuous video as the input. Experimentally, MPEG-1 compressed 
video has been utilized and a sequence of DC images generated using the approach described in B.-L. Yeo and B. Liu, 
"On the Extraction of DC Sequence from MPEG Compressed Video," Proc. of ICIP, October 1995. Both the pixel-based 
and the distribution-based difference metric are then computed from these DC images (difference metric collection), 

35 while video contents are measured to provide up-to-date test criteria (adaptation). Information from both the distribu- 
tion-based and pixel-based difference metric are fused (detection), after taking into account the new test criteria (signif- 
icance level) provided by the adaptation step. Finally, a list of scene change frame candidates is produced and filtered, 
resulting in the final scene change frame list (decision). 

DC images are reduced images formed from the collection of scaled DC coefficients in intra-coded DCT com- 

40 pressed video. Such images can be directly reconstructed from JPEG or movie^JPEG videos. For MPEG and H.261 
videos, fast reconstruction of DC images from motion-compensated pictures requires some levels of approximation. 
The technique described in B.-L Yeo and B. Liu, "On the Extraction of DC Sequence from MPEG Compressed Video," 
Proc. of ICIP, October 1995 is herein applied to generate DC luminance images from every single frame of MPEG video, 
while the chrominance components are simply discarded. Briefly, technique extracts the DC image corresponding to 

45 motion-compensated pictures directly from the compressed stream in the following manner. The technique first locates, 
for each block of interest, four original neighboring blocks from which the current block of interest is derived. It then 
approximate the DC term of each block of interest by the weighted sum of the DCs of the four original neighboring 
blocks, where the weights are simply the fraction of area occupied by the current block. 

It is noted, however, that this technique is not necessary in the realization of the present invention. Alternatively, one 

so can take uncompressed video frames and subsample them to create the same size of images to feed into the apparatus 
in accordance with the invention. 

In this step, both pixel-based and distribution-based difference metrics are collected as shown in Figure 8, which 
shows the sub-diagram of the difference metric collection step. In accordance with the invention, the pixel-based differ- 
ence metric for each frame t is the sum of absolute frame difference, 

55 

r " ij 
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where f\ represents the intensity value at pixel location (/,/) in frame 1 1 . 

The distribution-based difference metric is the traditional Kolmogorcv-Smirnov test metric, except that one each is 
computed herein for the entire image as well as its four equally divided sub-regions. 

5 Adaptation 

The purpose of this step is to provide the detection scheme with the ability to automatically adjust to different video 
contents. Figure 9 shows the sub-diagram of the adaptation step. As shown in Figure 9, statistics from each DC image 
and each pair of DC images are first collected to represent the current video content. They are the image contrast and 
w the motion estimate. The image contrast estimate is computed based on a recursive scheme to suppress the influences 
of sudden lighting changes. 

contrast t ={1 -x) contrast t ^ +xct M 

is where a M is the intensity variance estimate of the DC image at time M and x equals 0.6 in the present exemplary 
embodiment. 

Similarly the motion estimate is computed as follows 



20 

motion t =(1 -x) motion , A +x£ ( ^ ^ ) 

'j 

where f]j 1 is the intensity value at pixel location (/j) of the DC image at time M , N is the size of the image, 

25 and x equals 0.6 in the present exemplary embodiment. 

Both the image contrast and the motion estimates are then taken into a fuzzy engine to compute a new significance— 
level for the hierarchical Kolmogorov-Smirnov test. The fuzzy engine uses a quadratic membership function, where 
each contrast measurement is divided into four classes (low, middle, high, and extremely high), each motion estimate 
into three classes (slow, middle, and fast), and each significance level (K a ) into five classes (high, middle high, middle, 

so middle low, and low), as shown in Figures 10, 1 1 , and 12, illustrating the membership functions of the fuzzy engine in 
accordance with the invention. Figures 10, 1 1 , and 12 show the membership functions of contrast estimate, motion esti- 
mate, and K a which is proportional to significance level, respectively. 

Based on definitions of membership functions, the fuzzy rules are stated in a simple IF/THEN format, where values 
are combined using AND (minimum) or OR (maximum) operations. All rules are listed in the following Table 1 . 

35 

TABLE 1 

IF contrast is low and motion is slow, THEN significance level is middle 
40 IF contrast is middle and motion is slow THEN significance level is middle low 

IF contrast is high and motion is slow THEN significance level is middle low 

IF contrast is extremely high and motion is slow THEN significance level is low 

IF contrast is low and motion is fast THEN significance level is high 
45 IF contrast is middle and motion is fast THEN significance level is middle high 

IF contrast is high and motion is fast THEN significance level is middle 

IF contrast is extremely high and motion is fast THEN significance level is middle low 

50 IF contrast is low and motion is middle THEN significance level is high IF contrast is middle and motion is middle 
THEN significance level is middle high 

IF contrast is high and motion is middle THEN significance level is middle high 

IF contrast is extremely high and motion is middle THEN significance level is middle 



Since all rules have given different values for the significance level, they must be resolved or defuzzified to yield a 
crisp final output value. Here, the centroid method of Zadeh et al. is used to find the center of gravity of the combined 
output shape, ensuring all rules contributing to the final crisp result For the centroid method, see L. A. Zadeh and J. 
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Kacprzyk, Fuz2y Logic for the Management of Uncertainty, John Wiley & Sons, Inc., 1992. It is noted that although the 
above fuzzy membership functions and rules serve the purpose of the invention, these membership functions nor fuzzy 
rules have not been optimized. 

s Detection 

In this detection step, shown in the sub-diagram of Figure 13, both pixel-based and distribution-based difference 
metrics are independently tested and the results are fused to output scene change frame candidates. Pixel-based dif- 
ference metrics are treated as time series signals, where both visually abrupt cuts and the duplication of frames create 

w observation outliers obeying Equation (8). 

In time trend estimation, a piecewise linear function is used to model the trend, as opposed to a more sophisticated 
model. Robust statistics also utilized to avoid the influence of type II observation outliers in trend estimation (see the 
definitions previously presented). The difference metrics associated with frames from three frames before until three 
frames after the current frame are sorted. Assume they are ordered, d t < • • • d 7 . d 5 and d e are used to fit a line. 

75 The interpolated/extrapolated value at the current frame is the predicted value from the trend analysis. This process is 
shown in Figure 14, which shows an example of trend estimation (left) where a type I observation outlier is detected. 
Note that type II observation outliers exist in every other frame, wherein are plotted two sets of pixel-based difference 
metric (for example, sum of absolute difference against time. In both examples, frame 5 is the current frame, which is 
associated with a visually abrupt cut and a duplicated frame in the first and the second example respectively. 

20 First, samples are collected from frame 2 to frame 8 and sorted out in ascending order. Four smallest values (sam- 
ples from frame 2, 4, 6. 8 and from frame 3, 5, 6, 7 in the first and the second example respectively) are then discarded. 
This is because in any 7 frame interval the most type II observation outliers that one could have are four. In addition, 
the biggest value is also discarded (sample from frame 5 and from frame 4 in the first and the second example) because 
it may correspond to the visually abrupt cut as illustrated in Figure 14. 

25 Finally, samples from frame 3, 7 (in the first example), and from frame 2, 8 (in the second example) are chosen to 
fit a straight line. In the first example, the interpolated value at the current frame is much smaller than the actual value 
(positive deviation from the trend), whereas the interpolated value is much bigger than the actual value (negative devi- 
ation from the trend) in the second case. The former will be defined as type I observation outlier in the context of cut 
detection and the latter, type II observation outlier. 

30 After the time trend is estimated, it is subtracted from the observed time series and then test the hypothesis, 
H Q :A=0 , against the alternative, H-j:A*0 in Equation (8). A practical solution is obtained by considering a simple crite- 
rion of the form 



where A is the estimate of the displacement in the <?th observation and d - is the estimate of the variance of A . d - 
can be estimated by spectral methods (see U. Grenander and M. Rosenblatt, Statistical Analysis of Stationary Time 
40 Series, New York: Wiley, 1966, or, in accordance with the present exemplary embodiment of the invention, by substitut- 
ing usual estimates of a z and a r (r=1 ,2, • • • ,p) into 



45 



i+a?+-+ap 



Note that a r (r=1 ,2, • • • ,p) and a z are the same as in Equation (9). For performance reasons in the present exemplary 
so embodiment of the invention, a zeroth th -order autoregressive model is assumed, and therefore, a ,=0 (r=1 ,2, • • • ,p) 

Finally, for alt samples that are identified as observation outliers, if the predicted value from the trend is less than 
the observed, the frame is marked as a possible shot change frame candidate. 

55 Variance Estimate 

For those observed samples which are neither type I nor type 1 1 observation outliers, they are utilized to update the 
variance estimate a z using the following formula: 
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a z ={\-z)a z +x(frame difference signal with trend removed] 

where t equals 0.6. 

5 Hierarchical Kolmogorov-Smirnov test 

As described in previous sections, the Kolmogorov-Smirnov test is concerned with the agreement of two sets of 
observed values. The null hypothesis is that two samples come from populations with the same distribution function 
F(x) F(x) . It can be shown that 

w 

Prob{d,>K a ^a 

15 where K a represents a constant depending on the level of significance a and is available in tabular form from most sta- 
tistics books. See, for example, J. D. Gibbons and S. Chakraborti, Nonparametric Statistical Inference, Marcel Dekker, 
Inc., 1992. N is the size of the image the statistic is collected upon. d t , defined to be 

max}EDF t (j)-EDF t .i(j)\. 

20 

is constructed directly from the histogram of each DC image (see Equation (4). 

In each frame, a new significance level a is acquired from the adaptation step, a is used in the test involved not only 
the entire image but also each of its four equally divided sub images. Assume that n out of 4 regions show changes. 
The integrated count is defined as follows. 

25 

JCS t =n*2+l if the statistics based on the entire image 

3o shows a change} (12) 

KS c =n*2 otherwise} 



35 

This integrated count weights the change detected from any one of 4 regions twice more than the change detected 
from the entire image. 

In this stage, the integrated count produced by the Hierarchical Kolmogorov-Smirnov test is used to eliminate false 
positives produced by the time series outlier detection process. Denote the integrated count produced for frame f by <D, 

40 . The system eliminates a scene change frame candidate if 

When the system reaches this step, most false positives will have been eliminated. However, there could still be 
some due to flash light or extremely fast camera motion. A criterion can be set up to eliminate such false positives. The 
list of detected cut frames is stored in a delay memory as shown by the dashed box attached to the decision step in 
Figure 7 and examine the distance between any two cuts. Any cut that occurs in a distance less than a preset value (in 

45 the present exemplary embodiment, five frames) will be eliminated. 

The proposed cut browsing apparatus contains three basic elements: the Video Player, the Video Browser, and the 
Cut Browser. These three elements provide video browsing at three different levels of abstraction. The video player and 
the cut browser both present only raw video material, although in the latter case, the presented materials are at a dif- 
ferent level of abstraction. However, the video browser presents interpreted information about shots in the video. 

so Together, these three components offer the best tool to any user who would like to perform the following three tasks: to 
search the content of the video, to decide if the cut detection method may have missed or misdetected a shot, and to 
create or eliminate cuts. 

The proposed cut browsing apparatus works as follows. When it is started, three windows display on the screen as 
shown in Figure 15, which shows the cut browsing apparatus. 
55 The video browser shows all detected shots using representative frames (Rframes) , while the cut browser displays 
the cross section image made from the entire video. The activities in all three windows are completely synchronized. 
For example, if the user use the mouse to click on any point in the cut browser, the associated representative frame in 
the video browser will be highlighted and the video player will start playing back the video from the same point. When- 
ever the video browser gets scrolled, the cross-section image scrolls accordingly. 
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To carry out the first function, namely to search the content of the video, the user would examine each representa- 
tive frame in the video browser to see whether it is the desired shot For every shot the user would also examine the 
corresponding cross-section image in the cut browser to see if there is any pattern of missed shots. When any point is 
in doubt, the user would click on the point in the cut browser to look at the raw video playback in the video player 

To decide if the cut detection method may have missed or misdetected a shot, the user could simply examine the 
cross-section image in the cut browser. If there is any pattern of missed or misdetected shots, the user would then click 
on the point in the cut browser to look at the raw video playback in the video player. 

The cut browser also provides the editing capability. A user can break any shot into two or merge any two shots into 
one through a mouse or a button click. When this event happens, the cross-section image will be segmented differently 
and the list of representative frames will get updated as well. 

***The cut detection algorithm in accordance with the invention was applied to 24 video clips which included more 
than 149,000 MPEG-1 PAL and NTSC video frames. These videos come in a variety of types including movie trailers, 
cartoons, and sports videos, as listed in Table 2, following. Table 2 also shows the number of abrupt cuts (A/ c ) , the 
number of missed cuts (NJm}) (N m ) , and the number of false detected cuts (N f ) . These are experimental results from 
the automatic cut detection method in accordance with the invention. For those shots that are missed or misdetected, 
they all show clear patterns in cross-section images and therefore are easily identified manually using the present cut 
browsing apparatus. 

The present invention is believed to provide a recall and precision rate in the order of 99% and can detect from 
reading the video to the output of shot, change frames, close or near the video rate. The cut browsing apparatus has a 
cut browsing apparatus capable of substantially achieving 100 % recall rate. The invention takes into account the non- 
stationary nature of the cut detection problem, and has been tested on a variety of videos and over 149,000 video 
frames. 

Clearly, the present invention can be implemented in conjunction with the use of a suitably programmed digital com- 
puter, using programming techniques known in the art. 

While the present invention has been described by way of exemplary embodiments, these are not intended to be 
limiting and various changes and modifications will be apparent to those skilled in the art to which it pertains. For exam- 
ple, various details can be rearranged in the screens of the browsers without departing from the spirit of the invention. 
Furthermore, various alternative slicing planes for deriving cross section images can be substituted in a manner con- 
sistent with the invention. These and like changes are intended to be within the scope and contemplation of the inven- 
tion which is defined by the claims following. 

Claims 

1 . Cut browsing and editing apparatus for video images, comprising: 

a video player; 

a video browser, coupled to said video player; and 
a cut browser coupled to said video browser. 

2. Cut browsing apparatus as recited in Claim 1, wherein said video player and said cut browser present only raw 
video material. 

3. Cut browsing apparatus as recited in Claim 2, wherein said cut browser presents material at a different level of 
abstraction to that presented by said video player and said video browser presents interpreted information about 
shots in said video images. 

4. Cut browsing apparatus for video images, comprising: 

a video player; 

a video browser, coupled to said video player; 
a cut browser coupled to said video browser; and 

a display, coupled to the foregoing, said display providing a three-window display on a screen. 

5. Cut browsing apparatus as recited in Claim 4, wherein each window of said three-window display is respectively 
allocated to one of said ideo player, video browser, and said cut browser. 

6. Cut browsing apparatus as recited in Claim 5, wherein said windows respond synchronously activities including 
computer mouse pointer control, scrolling, and high-lighting. 
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7. Cut browsing apparatus as recited in Claim 4, wherein said video browser displays detected shots using represent- 
ative frames (Rframes), and said cut browser displays a cross sectional image made from an entire video 
sequence. 

s 8. Cut browsing apparatus as recited in Claim 7, wherein a user can select, as by mouse clicking, a point on said cross 
sectional image on said cut browser so as to view a corresponding image in raw video playback on said video 
player. 

9. Cut browsing apparatus as recited in Claim 8, wherein said cut browser permits editing by a user so as to break a 
w shot into at two shots and so as to merge any two shots into one, as by clicking a mouse. 

10. Cut browsing apparatus as recited in Claim 8, wherein said cut browser permits editing by a user so as to break a 
shot into a plurality of shots and so as to merge a number of said plurality of shots into one, as by clicking a mouse. 

15 11. Cut browsing apparatus as recited in Claim 9, wherein said editing causes resegmenting of said cross sectional 
image. 

12. Apparatus for detecting a cut in a video in accordance with claim 1 1 , including means for presenting two cross-sec- 
tion images of said video images, a horizontal cross section image in a horizontal direction and a vertical cross sec- 

20 tion image in a vertical direction of the video volume. 

13. Apparatus for detecting a cut in a video in accordance with claim 10, including means for presenting at least two 
cross-section images of said video images, whereof at least one shot is horizontal cross section image in a hori- 
zontal direction and at least one is vertical cross section image in a vertical direction of the video volume. 

25 

14. Cut browsing apparatus for video images, comprising: 

a video player; 

a video browser, coupled to said video player; 
so a cut browser coupled to said video browser; and 

a display, coupled to the foregoing, said display providing a three-window display on a screen, wherein each 
window of said three-window display is respectively allocated to one of said ideo player, video browser, and 
said cut browser; 

said windows respond synchronously activities including computer mouse pointer control, scrolling, and high- 
35 lighting; 

said video browser displays detected shots using representative frames (Rframes). and said cut browser dis- 
plays two cross-section images of said video images, a horizontal cross section image in a horizontal direction 
and a vertical cross section image in a vertical direction of the video volume. 

40 15. Cut browsing apparatus for detecting a cut in a video in accordance with claim 12, wherein each cross-section 
image is constructed by sampling one row (or column) from every image, and reducing the amount of information 
from a two-dimensional image to two one-dimensional image strips. 

16, Cut browsing apparatus for detecting a cut in a video in accordance with claim 13, wherein said horizontal and ver- 
45 tical cross-section images are combined into one image segmented into at least two bands according to a list of 

detected shots, whereby a level of abstraction is presented that is just enough to reveal whether there is a missed 
or misdetected shot. 

17, Cut browsing apparatus for video images, comprising: 

50 

a video player; 

a video browser, coupled to said video player; 
a cut browser coupled to said video browser; and 

a display, coupled to the foregoing, said display providing on a screen respective window displays for each of 
55 said video player, video browser, and cut browser, such that activities in all three are completely synchronized. 

18, Cut browsing apparatus as recited in Claim 17, wherein said windows respond synchronously activities including 
computer mouse pointer control, scrolling, and high-lighting. 
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19. Cut browsing apparatus as recited in Claim 18, wherein a user can select an image, as by mouse clicking, to view 
on sad cut browser an image corresponding to a selected raw video image on said video player. 

20. Cut browsing apparatus for video images, comprising: 

a video player; 

a video browser, coupled to said video player; 
a cut browser coupled to said video browser; and 

a display, coupled to the foregoing, said display providing a three-window display on a screen, wherein each 
window of said three-window display is respectively allocated to one of said ideo player, video browser, and 
said cut browser; 

said windows respond synchronously activities including computer mouse pointer control, scrolling, and high- 
lighting; 

said video browser displays detected shots using representative frames (Rframes), and said cut browser dis- 
plays a plurality of cross-section images of said video images, whereof one is a horizontal cross section image 
in a horizontal direction and one is a vertical cross section image in a vertical direction of the video volume. 

21 . Cut browsing apparatus for detecting a cut in a video in accordance with claim 20, wherein each cross-section 
image is constructed by sampling one row (or column) from every image, and reducing the amount of information 
from a two-dimensional image to a plurality of one-dimensional image strips. 

22. Cut browsing apparatus for detecting a cut in a video in accordance with claim 20, wherein said cross-section 
images are combined into one image segmented into a plurality of bands according to a list of detected shots, 
whereby a level of abstraction is presented that is enough to reveal whether there is a missed or misdetected shot. 
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FIG. 5 A 
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FIG. 6 A 
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FIG. 9 



DC IMAGE (COMPRESSED/VIDEO) 
SUBSAMPLEO/SMOOTHED IMAGE (UNCOMPRESSED VIDEO) 



IMAGE STATISTICS COLLECTION 



MOTION AND CONTRAST 
ESTIMATE 



FUZZY ENGINE 



NEW SIGNIFICANCE 
LEVEL ESTIMATE 



29 



EP 0 780 844 A2 




30 



EP0 780 844 A2 




EP 0 780 844 A2 




32 



EP 0 780 844 A2 



CO 



LU 



CD 
I — I 
CO 



LU 
CC 
LU 



5 



UJ 
CD 



I 

CD 



i — 3: 
i — i 

LUI — 
DC CO 



-cC LU 

CD CD 



CO LU 



CO 
CO 



LU 

-X I — 
cc 



LU 












CD 






H— « 


as: 






i — i 




CD 






Ll 


U 


CO 
LU 




1 — 
LU 








LU 








CD 




CC. 


3Z 


LU 


LU 


• — 1 


1 — 
-X 


» — 1 
1 


CC 


3: 


1— 


•<£ 


i— i 


ZD 




i — 


CD 




CO 








LU 







CD 
LU 



CD 

3T 

CD 



CD 
CO 



CD 
> — l 
CO 



cc 



LU 

CO 



CD 
CO 



cj re 






i — ii — i 






LCI — 






» — 1 CO ^ 


CD 




ZUJ ^ 


31 




CD 


i cc 




►— • 1 






COLU 


CD DT 






1 — t CO 




3=LU 


zn i i — 




LU 1 


CD > CO 
CC CD LU 




3: 






—x ac i — 


CD 1 — 




CC CD 




LU CO 
i—t CD 


31 CO 
1— I LU 




=3 










CD 






LU CC 












1 — CO 




3: i 










g§ 


is 




CD CC 
CD 




CD CD 






UJ CD 






1 — ac 




1 


iOROV- 
E ENT 




CC CD 




NTEG 
HE K 






1 — « 1 — 









33 



EP 0 780 844 A2 




34 



EP0 780 844 A2 




